Optimization Algorithms

Convex optimization

• Convexity

Gradient descent

• Gradient Descent

Optimization challenges

Common challenges

• local minima
• saddle points
  • any location where all gradients of a function vanish but which is neither a global nor a local minimum
  • 

Challenges in DL

• Almost all optimization problems arising in deep learning are nonconvex
• Gradient Descent#Challenges and Solutions